BACKGROUND: Artificial intelligence (AI) is becoming increasingly important in healthcare. It is therefore crucial that today's medical students have certain basic AI skills that enable them to use AI applications successfully. These basic skills are often referred to as "AI literacy". Previous research projects that aimed to investigate medical students' AI literacy and attitudes towards AI have not used reliable and validated assessment instruments. METHODS: We used two validated self-assessment scales to measure AI literacy (31 Likert-type items) and attitudes towards AI (5 Likert-type items) at two German medical schools. The scales were distributed to the medical students through an online questionnaire. The final sample consisted of a total of 377 medical students. We conducted a confirmatory factor analysis and calculated the internal consistency of the scales to check whether the scales were sufficiently reliable to be used in our sample. In addition, we calculated t-tests to determine group differences and Pearson's and Kendall's correlation coefficients to examine associations between individual variables. RESULTS: The model fit and internal consistency of the scales were satisfactory. Within the concept of AI literacy, we found that medical students at both medical schools rated their technical understanding of AI significantly lower (MMS1 = 2.85 and MMS2 = 2.50) than their ability to critically appraise (MMS1 = 4.99 and MMS2 = 4.83) or practically use AI (MMS1 = 4.52 and MMS2 = 4.32), which reveals a discrepancy of skills. In addition, female medical students rated their overall AI literacy significantly lower than male medical students, t(217.96) = -3.65, p <.001. Students in both samples seemed to be more accepting of AI than fearful of the technology, t(745.42) = 11.72, p <.001. Furthermore, we discovered a strong positive correlation between AI literacy and positive attitudes towards AI and a weak negative correlation between AI literacy and negative attitudes. Finally, we found that prior AI education and interest in AI is positively correlated with medical students' AI literacy. CONCLUSIONS: Courses to increase the AI literacy of medical students should focus more on technical aspects. There also appears to be a correlation between AI literacy and attitudes towards AI, which should be considered when planning AI courses.
Students, Medical , Humans , Male , Female , Literacy , Cross-Sectional Studies , Artificial Intelligence , Attitude of Health Personnel , Surveys and Questionnaires
Serious games, as a learning resource, enhance their game character by embedding game design elements that are typically used in entertainment games. Serious games in its entirety have already proven their teaching effectiveness in different educational contexts including medical education. The embedded game design elements play an essential role for a game's effectiveness and thus they should be selected based on evidence-based theories. For game design elements embedded in serious games used for the education of medical and healthcare professions, an overview of theories for the selection lacks. Additionally, it is still unclear whether and how single game design elements affect the learning effectiveness. Therefore, the main aim of this systematic review is threefold. Firstly, light will be shed on the single game design elements used in serious games in this area. Second, the game design elements' underlying theories will be worked out, and third, the game design elements' effectiveness on student learning outcome will be assessed. Two literature searches were conducted in November 2021 and May 2022 in six literature databases with keywords covering the fields of educational game design, serious game, and medical education. Out of 1006 initial records, 91 were included after applying predefined exclusion criteria. Data analysis revealed that the three most common game design elements were points, storyline, and feedback. Only four underlying theories were mentioned, and no study evaluated specific game design elements. Since game design elements should be based on theories to ensure meaningful evaluations, the conceptual GATE framework is introduced, which facilitates the selection of evidence-based game design elements for serious games.
PROBLEM: Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students' performance on LLM-generated questions to questions developed by humans. APPROACH: The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT. OUTCOMES: The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly. NEXT STEPS: Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.
Educational Measurement , Humans , Educational Measurement/methods , Students, Medical/statistics & numerical data , Education, Medical, Undergraduate/methods , Education, Medical/methods , Language , Female , Male
Tobacco smoking is the greatest preventable health risk. The effects are serious, both individually and societal. Nevertheless, the current prevalence of tobacco smokers in Germany is still high at around 35â%. A recent strong increase in actively smoking adolescents (14- to 17-year-olds, current prevalence approx. 16â%) and young adults (18- to 24-year-olds, current prevalence approx. 41â%) is also a cause for concern. About a third of all inpatients continue smoking while being treated. The hospitalization of active smokers in acute and rehabilitation hospitals serves as a "teachable moment" for initiation of cessation offers. An intervention that begins in hospital and continues for at least a month after discharge results in about 40â% additional smokefree patients. It is scientifically well-researched, effective and cost-efficient. After initiation in hospital these measures can be continued via ambulatory cessation programs, rehabilitation facilities, an Internet or telephone service. In Germany, there are structured and quality-assured cessation offers, both for the inpatient and for the outpatient area. The biggest obstacle to broad establishment of such offers is the lack of reimbursement. Two feasible ways to change this would be the remuneration of the existing OPS 9-501 "Multimodal inpatient treatment for smoking cessation" and the establishment of quality contracts according to §â110a SGB V. An expansion of tobacco cessation measures in healthcare facilities would reduce smoking prevalence, associated burden of disease and consecutive costs.
Smoking Cessation , Adolescent , Young Adult , Humans , Smoking Cessation/methods , Inpatients , Outpatients , Smoking/epidemiology , Delivery of Health Care
BACKGROUND: Scoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data. OBJECTIVE: This study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6). METHODS: Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05). RESULTS: We analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977). CONCLUSIONS: Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees' scores and overall examination results.
Serious games (computer-based learning games) are increasingly used in medical education at various levels, as user access is independent of location and time and promotes non-linear learning. In legal medicine, interactive digital media are still scarce. The freely accessible online serious game "Adventure Legal Medicine" was developed as part of the "Hamburg Open Online University". The goal was to teach the basics of forensic casework in a point-and-click adventure setting consisting of five cases. During development, 40 medical students were asked to evaluate the game anonymously. The System Usability Scale (SUS) resulted in a mean score of 86.7 (SD 8.3), which corresponds to above-average usability. Further specific evaluations revealed a good to very good rating of the game with no differences in terms of gender (p = 0.214), first-year versus advanced students (p = 0.393) and students who never/rarely or sometimes/often played computer games (p = 0.780). Since there are only a few digital media so far that allow curricular integration into undergraduate teaching in legal medicine, this serious game represents a possibility to integrate digital media into both face-to-face teaching and distance learning and to use it as a supplement to the medical school's own teaching offer, encouraging users to actively engage with the subject.
Education, Medical, Undergraduate , Education, Medical , Humans , Education, Medical, Undergraduate/methods , Internet , Learning , Education, Medical/methods , Forensic Medicine
PURPOSE: Medical education has been transformed during the COVID-19 pandemic, creating challenges regarding adequate training in ultrasound (US). Due to the discontinuation of traditional classroom teaching, the need to expand digital learning opportunities is undeniable. The aim of our study is to develop a tele-guided US course for undergraduate medical students and test the feasibility and efficacy of this digital US teaching method. MATERIALS AND METHODS: A tele-guided US course was established for medical students. Students underwent seven US organ modules. Each module took place in a flipped classroom concept via the Amboss platform, providing supplementary e-learning material that was optional and included information on each of the US modules. An objective structured assessment of US skills (OSAUS) was implemented as the final exam. US images of the course and exam were rated by the Brightness Mode Quality Ultrasound Imaging Examination Technique (B-QUIET). Achieved points in image rating were compared to the OSAUS exam. RESULTS: A total of 15 medical students were enrolled. Students achieved an average score of 154.5 (SD ±â11.72) out of 175 points (88.29â%) in OSAUS, which corresponded to the image rating using B-QUIET. Interrater analysis of US images showed a favorable agreement with an ICC (2.1) of 0.895 (95â% confidence interval 0.858â< ICC <â0.924). CONCLUSION: US training via teleguidance should be considered in medical education. Our pilot study demonstrates the feasibility of a concept that can be used in the future to improve US training of medical students even during a pandemic.
COVID-19 , Students, Medical , Humans , Thyroid Gland , Pilot Projects , Feasibility Studies , Pandemics , Abdomen , Ultrasonography , Thorax , Curriculum
Aims and objectives: Digital teaching, learning and assessment have been part of medical education and continuing education for decades. The objective of this review paper is to highlight developments and perspectives in these areas in the GMS Journal for Medical Education (GMS JME). Methodology: In the spring of 2020, we conducted a systematic literature search of the Journal for Medical Education (JME) and analysed the articles with regard to different categories such as article type, digital tools used or mode of data collection. Results: Of the 132 articles analysed, 78 were digital interventions (53 of which were exploratory-descriptive), 28 were project descriptions, 16 were surveys of needs or equipment and 10 were concept papers. About one-third of the studies and project reports each dealt with virtual patients or case-based learning, whereas no articles were published on trends such as serious games or virtual reality. Overall, our analysis shows that in many respects, the studies on digital teaching were more broadly based, especially between 2006 and 2010, after which this trend tended to decline again. Conclusions: Our analysis shows that publications in the JME consider some key aspects of digital teaching in medical education and continuing education, such as educational videos or virtual patients. The variability of information and methods of presentation advocate the use of guidelines to optimise the quality of scientific papers. Furthermore, clues for future research topics and experimental study designs are identified.
Education, Medical , Learning , Humans
This randomized crossover trial examines whether elaboration on common errors in patient treatment, combined with individualized mailed feedback, improves medium-term retention of clinical reasoning competence.
Students, Medical , Humans , Clinical Reasoning , Feedback , Clinical Competence , Problem Solving
BACKGROUND: The use of artificial intelligence applications in medicine is becoming increasingly common. At the same time, however, there are few initiatives to teach this important and timely topic to medical students. One reason for this is the predetermined medical curriculum, which leaves very little room for new topics that were not included before. We present a flipped classroom course designed to give undergraduate medical students an elaborated first impression of AI and to increase their "AI readiness". METHODS: The course was tested and evaluated at Bonn Medical School in Germany with medical students in semester three or higher and consisted of a mixture of online self-study units and online classroom lessons. While the online content provided the theoretical underpinnings and demonstrated different perspectives on AI in medical imaging, the classroom sessions offered deeper insight into how "human" diagnostic decision-making differs from AI diagnoses. This was achieved through interactive exercises in which students first diagnosed medical image data themselves and then compared their results with the AI diagnoses. We adapted the "Medical Artificial Intelligence Scale for Medical Students" to evaluate differences in "AI readiness" before and after taking part in the course. These differences were measured by calculating the so called "comparative self-assessment gain" (CSA gain) which enables a valid and reliable representation of changes in behaviour, attitudes, or knowledge. RESULTS: We found a statistically significant increase in perceived AI readiness. While values of CSA gain were different across items and factors, the overall CSA gain regarding AI readiness was satisfactory. CONCLUSION: Attending a course developed to increase knowledge about AI in medical imaging can increase self-perceived AI readiness in medical students.
Students, Medical , Humans , Literacy , Artificial Intelligence , Curriculum , Schools, Medical
Background: Due to the coronavirus pandemic, the medical faculties in the Federal Republic of Germany converted their curricula to digital formats on a large scale and very quickly in spring 2020 as an emergency measure. At the same time, a start was made on the nationwide exchange of digital teaching/learning materials via the online platform "LOOOP share" in order to save local resources. Among other things, virtual patient cases (VP) were shared across faculties for case-based learning, through which students can acquire clinical decision-making skills. Objectives: Within the framework of the cooperation project "National Learning Platforms for Digital Patient-Related Learning in Medical Studies" (DigiPaL), the usability of VPs for students and teachers should be improved, and the spectrum of disease patterns that are covered by VPs should be systematically expanded. Results: With the participation of many locations, a total of 150 VPs were developed by 96 case authors from 16 faculties, in addition to the existing 403 VPs. The thematic selection was made on the basis of criteria oriented to the National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM). After completion, these VPs were also made available to all faculties for free use via "LOOOP share" and the CASUS learning platform. Discussion: Even after the pandemic, these developed VPs should be available to the faculties and thus make a lasting contribution to improve medical training in Germany - especially in light of digital teaching formats being expressly advocated on the basis of the adapted current Medical Licensure Act (ÄApprO). A possible application is interdisciplinary learning of clinical decision-making with the help of blended learning formats within the framework of a longitudinal curriculum. The large number of involved colleagues and faculties shows that the nationally coordinated development of VPs across faculties was commonly seen as useful.
Education, Medical, Undergraduate , Humans , Curriculum , Faculty, Medical , Learning , Licensure, Medical , Clinical Competence , Germany
Prior work experience in a relevant medical profession is an important admission criterion currently used at many German medical schools in addition to cognitive criteria. In other countries, work experience is often considered in later admission stages (e.g., interviews with pre-selected subgroups of applicants). However, evidence for its predictive validity for study success in addition to cognitive admission criteria is currently lacking. We therefore assessed whether completed vocational training in a relevant medical profession can predict study performance in the first two years of study in addition to cognitive admission criteria. Admission and study performance data of all currently enrolled medical students at two German medical schools (Göttingen and Heidelberg) beginning with the 2013/14 cohort were retrospectively analyzed. Cognitive admission criteria in our sample were GPA grades and a cognitive test ("Test für Medizinische Studiengänge", TMS). We defined the study outcome parameter as the mean percentile rank over all performance data points over the first two years of study for each location, respectively. A multi-level model with varying intercepts by location, GPA, TMS, vocational training, and sex as predictors accounted for 14.5% of the variance in study outcome. A positive predictive association with study outcome was found for vocational training (ß = 0.33, p = .008) beyond GPA (ß = 0.38, p < .001) and TMS (ß = 0.26, p < .001). Our results support the use of prior vocational training as a selection criterion for medical studies potentially adding predictive validity to cognitive criteria.
Students, Medical , Humans , Students, Medical/psychology , School Admission Criteria , Vocational Education , Retrospective Studies , Schools, Medical , Educational Measurement
OBJECTIVE: Clinical reasoning is an essential skill, the foundations of which should be acquired during medical school. Within the format of test-based learning, such examinations can also be used to support the long-term retention of procedural knowledge necessary for clinical reasoning. The aim was to investigate whether repeated exposure to clinical cases in obstetrics and gynecology (OBGYN) with built-in questions leads to higher learning outcome than pure reading cases and what influence the delay between the intervention and the final test has on the retention of the respective content. METHODS: In this non-randomised crossover study, 5th-year medical students (duration of the study is 6 years) taking a 1-week clinical attachment in OBGYN participated in computer-based case seminars in winter term 2020/2021, in which different case histories on gynecological-obstetric diseases were presented. Case content was identical for all groups, but the presentation format (cases with key feature questions vs read-only cases) of individual case vignettes changed weekly. The also intervention was repeated after 2 weeks for each group. Knowledge was assessed in an entry and an exit exam consisting of 40 short-answer questions. RESULTS: A total of 94 out of 118 eligible students participated in the study (response rate: 79.7%). Learning outcome was significantly higher for items presented in the key feature format compared to items presented as read-only cases (74.2 ± 8.6% vs. 71.0 ± 9.2%; p = 0.017). Furthermore, the analysis showed that the temporal distance of the intervention package from the final examination had no influence on retention. CONCLUSION: This is the first study to demonstrate an effect of test-enhanced learning on clinical reasoning in the subject of OGBYN. In this cross-over study, repeated testing was more effective than repeated case-based learning alone. Curricular implementation of longitudinal key feature testing can thus improve learning outcomes for OBGYN.
Education, Medical , Gynecology , Obstetrics , Students, Medical , Cross-Over Studies , Education, Medical/methods , Educational Measurement , Gynecology/education , Humans , Learning , Obstetrics/education , Prospective Studies , Students, Medical/psychology
BACKGROUND: Validation of examinations is usually based on classical test theory. In this study, we analysed a key feature examination according to item response theory and compared the results with those of a classical test theory approach. METHODS: Over the course of five years, 805 fourth-year undergraduate students took a key feature examination on general medicine consisting of 30 items. Analyses were run according to a classical test theory approach as well as using item response theory. Classical test theory analyses are reported as item difficulty, discriminatory power, and Cronbach's alpha while item response theory analyses are presented as item characteristics curves, item information curves and a test information function. RESULTS: According to classical test theory findings, the examination was labelled as easy. Analyses according to item response theory more specifically indicated that the examination was most suited to identify struggling students. Furthermore, the analysis allowed for adapting the examination to specific ability ranges by removing items, as well as comparing multiple samples with varying ability ranges. CONCLUSIONS: Item response theory analyses revealed results not yielded by classical test theory. Thus, both approaches should be routinely combined to increase the information yield of examination data.
Clinical Reasoning , Educational Measurement , Humans , Educational Measurement/methods , Psychometrics
Purpose: Physicians of all specialties must be familiar with important principles of infectious diseases, but curricular time for this content is limited and clinical teaching requires considerable resources in terms of available patients and teachers. Serious games are scalable interventions that can help standardize teaching. This study assessed whether knowledge and skills acquired in a serious game translate to better performance in a clinical examination. Methods: Fifth-year undergraduate medical students (n = 100) at Goettingen Medical School were randomized to three groups receiving different levels of exposure to virtual patients presenting with signs and symptoms of either infective endocarditis or community-acquired pneumonia in a serious game simulating an accident and emergency department. Student performance was assessed based on game logfiles and an objective standardized clinical examination (OSCE). Results: Higher exposure to virtual patients in the serious game did not result in superior OSCE scores. However, there was good agreement between student performance in the OSCE and in game logfiles (r = 0.477, p = 0.005). An Item Response Theory analysis suggested that items from the serious game covered a wider range of ability, thus better differentiating between students within a given cohort. Conclusion: Repeated exposure to virtual patients with infectious diseases in a serious game did not directly impact on exam performance but game logfiles might be good and resource-sparing indicators of student ability. One advantage of using serious games in medical education is standardized content, a lower inhibition threshold to learn, and a need of less staff time compared to small-group clinical teaching.
BACKGROUND: The coronavirus pandemic has led to increased use of digital teaching formats in medical education. A number of studies have assessed student satisfaction with these resources. However, there is a lack of studies investigating changes in student performance following the switch from contact to virtual teaching. Specifically, there are no studies linking student use of digital resources to learning outcome and examining predictors of failure. METHODS: Student performance before (winter term 2019/20: contact teaching) and during (summer term 2020: no contact teaching) the pandemic was compared prospectively in a cohort of 162 medical students enrolled in the clinical phase of a five-year undergraduate curriculum. Use of and performance in various digital resources (case-based teaching in a modified flipped classroom approach; formative key feature examinations of clinical reasoning; daily multiple choice quizzes) was recorded in summer 2020. Student scores in summative examinations were compared to examination scores in the previous term. Associations between student characteristics, resource use and summative examination results were used to identify predictors of performance. RESULTS: Not all students made complete use of the digital learning resources provided. Timely completion of tasks was associated with superior performance compared to delayed completion. Female students scored significantly fewer points in formative key feature examinations and digital quizzes. Overall, higher rankings within the student cohort (according to summative exams) in winter term 2019/20 as well as male gender predicted summative exam performance in summer 2020. Scores achieved in the first formative key feature examination predicted summative end-of-module exam scores. CONCLUSIONS: The association between timely completion of tasks as well as early performance in a module and summative exams might help to identify students at risk and offering help early on. The unexpected gender difference requires further study to determine whether the shift to a digital-only curriculum disadvantages female students.
Students, Medical , Curriculum , Educational Measurement/methods , Female , Humans , Learning , Male , Pandemics , Prospective Studies , Teaching
INTRODUCTION: Observational and trial evidence conflict on the efficacy of two contrasting behavioral approaches to quitting smoking-gradual and abrupt. Observational data suggest an abrupt approach to quitting is superior to a gradual approach, whilst trials show no difference. One potential explanation is self-selection in observational data, whereby people can choose their quit approach, and those who find it harder to quit may be more likely to choose a gradual quit approach. This study aims to investigate potential explanations for these conflicting findings. AIMS AND METHODS: This study aims to investigate potential explanations for these conflicting findings. We used observational data from a nationally representative sample of adults in England from November 2006 to February 2020 who reported smoking and had made at least one quit attempt in the past year (n = 21 542). We used logistic regression models to assess the association between abrupt versus gradual quit attempts and quit success, adjusting for sociodemographic, smoking, and quit attempt characteristics. FINDINGS: Abrupt, versus gradual, attempts were associated with improved quit success in an unadjusted model (odds ratio = 2.02, 95% CI = 1.86 to 2.19). This association remained after adjusting for a broad range of relevant confounders (odds ratio = 1.75, 95% CI = 1.59 to 1.93). CONCLUSIONS: Among a representative sample of adults who had smoked and made a quit attempt in the past year, there was evidence of an association between abrupt attempts and quit success before and after adjusting for relevant confounders. This suggests that the differences in quit success seen between abrupt and gradual quit attempt types are not completely driven by self-selection in observational data. IMPLICATIONS: We investigated explanations for conflicting findings on the efficacy of gradual versus abrupt approaches to quitting smoking between trial and observational data. Despite adjusting observational data for sociodemographic, smoking, and quit attempt characteristics, an association between abrupt quitting and quit success remained. Therefore, differences in quit success were not completely driven by the self-selection of a gradual approach by people who found it especially difficult to quit or differences in the use of quitting aids. However, characteristics adjusted for were limited by the data available, and future research should continue to investigate the difference in findings across study types to inform cessation support.
Smoking Cessation , Adult , England/epidemiology , Health Behavior , Humans , Smoking/epidemiology , Tobacco Use Cessation Devices
